Temporal alignment system and method

ABSTRACT

A temporal alignment system and method for example for detecting temporal misalignment in video frames when the frames are divided for transport using a signal divider for dividing a single signal S into portions S 1  . . . S N  and using average picture level in determining whether data sets within a particular frame are misaligned.

PRIORITY CLAIM AND INCORPORATION BY REFERENCE

This application is a continuation of U.S. patent application Ser. No.15/261,895 filed Sep. 10, 2016 and claims the benefit of U.S.Provisional Patent Application No. 62/217,523 filed Sep. 11, 2015,entitled Temporal Alignment System and Method, which are incorporatedherein by reference in their entirety and for all purposes.

BACKGROUND OF THE INVENTION

Broadcast and production signal transport requiring original signaldivision and multilink transport is in a protracted infancy. Reasonsinclude lack of proven equipment and movie and television industryreluctance to develop, implement, and use equipment capable oftransporting ultra-high definition signals such as or similar to 4Ksignals (see e.g., DCI 4K, UHDTV, UHD-1, UHD 4K, 4K).

FIELD OF INVENTION

This invention relates to the electrical and process arts. Inparticular, a system and method for enhancing temporal signal alignment.

DISCUSSION OF THE RELATED ART

While temporal signal alignment is well known in some applications,temporal signal alignment and realignment in the context of 4K imagestransported over multiple lines is not well known. Further, temporalsignal alignment systems operating in a 4K environment without the useof time stamps embedded in the signal are to the author's knowledgeunknown.

SUMMARY OF THE INVENTION

The present invention provides a temporal alignment system and method.Various embodiments may be used in connection with determining whetherframes, panes, signals, and/or data sets are aligned.

In an embodiment, a system for detecting temporal misalignment in videoframes that are divided for transport, the system comprises: a signaldivider for dividing a single signal S into N signal portions S₁ . . .S_(N); each signal portion for carrying a portion of the same videoframe; N transport links, each link for receiving a respective signal Sxfrom the signal divider and for delivering a signal Sx′ to a signalreassembler (1≦x≦N); the signal reassembler for selectively reassemblingsignal portions S₁′ . . . S_(N)′ corresponding to S₁ . . . S_(N) into asingle signal S′; and, a misalignment monitor colocated with the signalreassembler; wherein the monitor utilizes information inherent in thesignal portions S₁′ . . . S_(N)′ to detect whether temporal framemisalignment has occurred.

In some embodiments, the system above and one or more of i) wherein thesignals are Serial Digital Interface signals for transmitting 4K video,ii) wherein N=4, iii) further comprising a facility of the monitor forevaluating average picture level for each of signal portions S₁′ . . .S_(N)′ and wherein the monitor facility for evaluating average picturelevel is used in determining whether a temporal frame misalignment hasoccurred and used in correcting a discovered temporal framemisalignment, iv) further comprising a facility of the monitor forevaluating average picture level for each of signal portions S₁′ . . .S_(N)′ and wherein changes in average picture level caused by one ormore scene cuts are used in determining whether a temporal framemisalignment has occurred, v) further comprising a facility of themonitor for evaluating average picture level for each of signal portionsS₁′ . . . S_(N)′ and wherein temporal alignment of changes in averagepicture level among the signal portions S₁′ . . . S_(N)′ is used todetermine whether a temporal frame misalignment has occurred, and vi)further comprising a facility of the monitor for evaluating averagepicture level for each of signal portions S₁′ . . . S_(N)′ and whereintemporal alignment of discontinuities in average picture level among thesignal portions S₁′ . . . S_(N)′ is used to determine whether temporalframe misalignment has occurred.

In an embodiment a video frame temporal misalignment detection methodcomprising the steps of: providing a 4K video camera for acquiringimages, each image contained in a video frame that is divisible into 4panes; from one or more camera outputs, deriving 4 SDI signalscorresponding to respective video frame panes; the SDI signalstransporting the video frames over respective links such that duringtransport a pane of a particular frame is temporally misaligned andappears as a pane of another frame; providing a misalignment monitorwith a facility for monitoring average picture level of each of the SDIsignals; and, determining a temporal misalignment event has occurredwhen average picture level changes fail to occur simultaneously.

In some embodiments, the method above and one or more of the steps i)wherein the monitor is configured to detect changes in average picturelevel arising from scene cuts, ii) wherein each pane of a frame is a onequarter slice of the image, wherein each pane of a frame reproduces theimage at a reduced resolution, and iii) providing a frame reassemblercoupled to the links and the monitor and wherein upon discovery of avideo frame temporal misalignment event the monitor causes thereassembler to correct the misalignment by time shifting reassemblerdata received from at least one of the derived SDI signals.

In an embodiment, method for finding misaligned data sets comprising thesteps of: dividing a 4K video signal for transport along 4 SDI links;transporting the 4K video signal along the SDI links; near a transportterminus, comparing temporal alignment of an average picture leveldiscontinuity in each SDI link; and, finding a temporal correction isneeded when the discontinuities are not substantially temporallyaligned.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described with reference to the accompanyingfigures. The figures, incorporated herein and forming part of thespecification, illustrate the present invention and, together with thedescription, further serve to explain the principles of the inventionand to enable a person skilled in the relevant art to make and use theinvention.

FIG. 1 shows a multi-link signal transport system of the presentinvention.

FIG. 2 shows a source with signal division for use with the system ofFIG. 1.

FIG. 3A shows a 4K image.

FIG. 3B shows a square division quad split of the image of FIG. 3A.

FIG. 4A shows a 4K image.

FIG. 4B shows an interleaved or two sample interleave of the image ofFIG. 4A.

FIG. 5 shows an exemplary multi-link signal transport system withtemporal alignment monitoring.

FIG. 6A illustrates temporally aligned signal transport 600A.

FIG. 6B shows an exemplary inherent temporal alignment assessmentmethod.

FIG. 7A illustrates temporally misaligned signal transport 700A.

FIG. 7B shows an exemplary inherent temporal misalignment assessmentmethod.

FIG. 8 shows another embodiment of the multi-link signal transportsystem with temporal alignment monitoring of FIG. 5.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The disclosure provided in the following pages describes examples ofsome embodiments of the invention. The designs, figures, anddescriptions are non-limiting examples of certain embodiments of theinvention. For example, other embodiments of the disclosed device may ormay not include the features described herein. Moreover, disclosedadvantages and benefits may apply to only certain embodiments of theinvention and should not be used to limit the disclosed inventions.

Where parts are connected, descriptions herein using the words“coupled,” “connected,” or “interconnected” refer to either direct orindirect connections. Direct connections provide for a first partconnected directly to a second part, for example A connected directly toB. Indirect connections provide for a first part connected indirectly toa second part, for example A connected indirectly to C via B.

FIG. 1 shows a multi-link signal transport system 100. As seen, atransport block 112 interconnects a source block 102 and a destinationblock 122.

In the source block 102, a signal such as an image signal is dividedamong four signal outputs s1-s4. In an embodiment, the source blockincludes a video camera that i) acquires a 4K or similar image and ii)provides multiple output signals s1-s4 that may be combined to reproducethe acquired 4K image.

Signals such as image signals transported from a source 102 to adestination 122, which may be and/or include a reassembler, can bedivided at the source, transported via links, and reassembled at thedestination. For example, where a 4K image is transported 112, it may betransported via multiple HD-SDI links. In particular, four (4) HD-SDIlinks 114 at 3 Gb/s might be used where each link carries one quarter(¼) of the 4K image information. As shown, signals s1-s4 from the source102 are transported via links 114 and arrive at the destination 122 assignals s1′-s4′.

Notably, to the extent that signals s1-s4 are transported 112 withtemporal and content fidelity, then signals s1′-s4′ may be reassembledto reproduce a 4K image that was acquired by the camera. However, it isnot always the case that fidelity is maintained. In particular, loss oftemporal fidelity may occur due, for example, to varying signal transittimes among the transport links 114.

FIG. 2 shows a source with signal division 200. Where content such as 4Kimagery is captured via a device such as a digital video camera, thecontent may be readied for transport by multiple links as describedabove. For example, imagery captured is a source capture block 202results in a signal “s” that is fed 203 to a source outputs block 212.The four square checkerboard icon with all four blocks marked indicatesa full bandwidth 4K signal.

In the source outputs block 212, the 4K signal is repackaged fortransport by four links. At a first output 213, a signal s1 provides ¼of the 4K bandwidth and carries ¼ of the 4K content as indicated by thetop left block of the checkerboard icon. At a second output 215, asignal s2 provides ¼ of the 4K bandwidth and carries ¼ of the 4K contentas indicated by the top right block of the checkerboard icon. At a thirdoutput 217, a signal s3 provides ¼ of the 4K bandwidth and carries ¼ ofthe 4K content as indicated by the bottom right block of thecheckerboard icon. At a fourth output 219, a signal s4 provides ¼ of the4K bandwidth and carries ¼ of the 4K content as indicated by the bottomleft block of the checkerboard icon.

Signals such as images and 4K images in 4K camera video can be carriedby multiple links as described above. FIGS. 3A-B, 4A-B illustrate twotypes of signal or image division enabling multilink transport.

FIG. 3A shows a 4K image 300A and FIG. 3B shows a square division quadsplit 300B of the image of FIG. 3A. In particular, the image of FIG. 3Ais divided into four image portions, each image portion reproducing orindicating a ¼ share of the original 4K image. Pane 1 is upper left,pane 2 is upper right, pane 4 is lower right, and pane 3 is lower left.As skilled artisans will appreciate, reproduction of the original 4Kimage from a group of 4 panes representing a single frame requirestemporal alignment of the panes.

FIG. 4A shows a 4K image 400A and FIG. 4B shows an interleaved or twosample interleave of the image 400B of FIG. 4A. As in FIGS. 3A-B, the 4Kimage is divided into 4 panes. But, here, each pane carries the wholeimage at ¼ of its original resolution. This method is referred to asinterleaving, here a 2 sample interleave. For example, pane 1 is thewhole image with 0-25% of the content, pane 2 is the whole image with26-50% of the content, pane 3 is the whole image with 51-75% of thecontent, and pane 4 is the whole image with 76-100% of the content.

FIG. 5 shows an exemplary multi-link signal transport system withtemporal alignment monitoring 500. As seen, a transport block 512interconnects a source block 502 and a destination block 522.

In the source block 502, a signal such as an image signal is dividedamong four signal outputs s1-s4. In an embodiment, the source blockincludes a video camera that i) acquires a 4K or similar image and ii)provides multiple output signals s1-s4 that may be combined to reproducethe acquired 4K image.

As described above, signals such as image signals transported from asource 502 to a destination 522 can be divided at the source,transported via links, and reassembled at the destination. Signalprocessing such as compression may occur at a first signal processor 511and signal processing such as decompression may occur at a second signalprocessor 513 (e.g., JPEG 2000, H264). In some embodiments, thecompression and decompression method provides for a lossless datatransfer.

Signals s1-s4 from the source 502 are transported via links 114 andarrive at the destination 522 as signals s1′-s4′. Notably, to the extentthat signals s1-s4 are transported 512 with temporal and contentfidelity, then signals s1′-s4′ may be combined to reproduce a 4K imagethat is a replica or near replica of the image acquired by the camera.However, it is not always the case that temporal fidelity is maintained.In particular, loss of temporal fidelity may occur due, for example, tovarying signal transit times among the transport links 114.

As shown, the transported signals s1′-s4′ are inputs to a monitor ormonitoring block 532. The monitoring block provides a means fordetermining temporal alignment and/or temporal misalignment of signalss1′-s4′. In some embodiments, the destination block 522 and monitorblock may be collocated and/or packaged in a common main (as in a rack)or sub (as in a chassis enclosure) housing. In some embodiments thepackaged destination 522 and monitor 532 blocks may be referred to as areceiver 542.

FIG. 6A illustrates temporally aligned signal transport 600A. As seen,signals s1-s4 load frames 1, 2 for transport 610. Each frame is dividedfor transport along four links 614.

At the transport input, frame 1 includes four temporally aligned datasets resulting from dividing a first signal such as a 4K signal. Frame 2includes 4 temporally aligned data sets resulting from dividing a secondsignal such as a 4K signal.

At the transport output, frame 1′ includes the four temporally aligneddata sets that were input as frame 1 and frame 2′ includes the fourtemporally aligned data sets that were input as frame 2.

Methods of monitoring the temporal alignment illustrated in FIG. 6Ainclude time stamp methods where information added, i.e., the “timestamp,” to the content being transported provides a basis for assessingtemporal alignment.

Another temporal alignment assessment method does not require thatinformation be added. Rather, information inherent in the contentprovides a basis for assessing temporal alignment. One or both ofinherent and non-inherent assessment methods may be used in the temporalalignment monitor 532 of FIG. 5.

Corresponding to FIG. 6A, FIG. 6B shows an exemplary inherent temporalalignment assessment method 600B. For each of four links and linksignals s1-s4 values of average picture level (“APL”) are plottedagainst a common time axis. In an embodiment, average picture level isan average level of the picture signal. In an embodiment, averagepicture level is the average level of the picture signal during activescanning time integrated over a frame period. APL may be defined as apercentage of the range between blanking and reference white level. Insome embodiments, any consistently used measure of APL suffices (seee.g., IDMS 1.03 Information Display Measurements Standard (IDMS),pre-gamma APL (Type 1 APL) for gamma corrected input signal(R, G, B) andpost-gamma APL (Type 2 APL) for gamma de-corrected panel displaysignal(R′, G′, B′)).

As the plots show, APL data on each of the links (APLs1 . . . APLs4)indicates a change or discontinuity in average picture level such as anAPL change associated with a scene cut. Moreover, this change in APLoccurs simultaneously, at time t1, for all of the links.

Monitoring this information, temporal alignment of changes ordiscontinuities, provides a method of finding when data on multiplelinks representing a particular frame is not temporally aligned.Notably, this telltale sign of temporal misalignment is part of thepicture content and thus is inherent in the signals being transported.

FIG. 7A illustrates a signal transport that is temporally misaligned700A. As seen, signals s1-s4 load frames 1, 2 for transport 710. Eachframe is divided for transport along four links 714.

At the transport input, frame 1 includes four temporally aligned datasets resulting from dividing a first signal such as a 4K signal. Frame 2includes 4 temporally aligned data sets resulting from dividing a secondsignal such as a 4K signal.

At the transport output, frame 1′ includes three temporally aligned datasets that were input as frame 1 and one temporally misaligned data set731. Frame 1′ is therefore temporally misaligned, for example becausethe frame excludes data it should include or includes input data setsfrom different input data frames. In similar fashion, frame 2′ istemporally misaligned, for example because it includes a misalignedframe 733.

Corresponding to FIG. 7A, FIG. 7B shows an assessment of temporalalignment 700B. For each of four links and link signals s1-s4 values ofaverage picture level (“APL”) are plotted against a common time axis.

As the plots show, APL data on each of the links (APLs1 . . . APLs4)indicates a change or discontinuity in average picture level such as anAPL change associated with a scene cut. As the plots show, APL data oneach of the links (APLs1 . . . APLs4) indicates a change ordiscontinuity in average picture level such as an APL change associatedwith a scene cut.

For three of the links, APLs1, APLs3, APLs4, the change in APL occurssimultaneously, at time t1 and corresponds with a first discontinuity,for example a scene cut, such as scene cut 1.

For one of the links, APLs2, the change in APL occurs at a differenttime t2 corresponding with a second discontinuity such as scene cut 2where t2−t1=dt.

Monitoring this information, temporal alignment of changes ordiscontinuities, provides a method of finding when data on multiplelinks representing a particular frame is not temporally aligned.Notably, this telltale sign of temporal misalignment is part of thepicture content and thus is inherent in the signals being transported.

FIG. 8 shows another embodiment 800 of the multi-link signal transportsystem with temporal alignment monitoring of FIG. 5.

In an embodiment, a first signal line 551 interconnecting the monitorand transport blocks 532, 512 enables the monitor block to realign datasets along links such that data sets within frames are aligned. Controlfunction capabilities include speeding or slowing transport throughputon any one or more of the links 514.

In an embodiment, a second signal line 553 interconnecting the monitorand source blocks 532, 502 enables the monitor block to realign datasets along links such that data sets within frames are aligned. Controlfunction capabilities including speeding or slowing source data ratesalong any one or more of the data source outputs.

The appendix to this application provides descriptions of similar and/orother embodiments of the present invention.

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample only, and not limitation. It will be apparent to those skilledin the art that various changes in the form and details can be madewithout departing from the spirit and scope of the invention. As such,the breadth and scope of the present invention should not be limited bythe above-described exemplary embodiments, but should be defined only inaccordance with the following claims and equivalents thereof.

What is claimed is:
 1. A method for finding misaligned data setscomprising the steps of: dividing a video signal for transport alongmultiple links; transporting the video signal along the links; near atransport terminus, comparing temporal alignment of an average picturelevel discontinuity in each link; and, finding a temporal correction isneeded when the discontinuities are not substantially temporallyaligned.
 2. The method of claim 1 wherein the video signal is a SerialDigital Interface (“SDI”) signal.
 3. The method of claim 2 wherein thevideo signal is a 4K signal.
 4. The method of claim 1 wherein: thedivided video signals result from an image conveyed by the video signalthat is divided into panes; and, the data associated with each pane istransported along a respective one of the multiple links.
 5. The methodof claim 4 wherein average picture level changes result from scene cuts.6. The method of claim 4 wherein each pane reproduces the entirety ofthe image conveyed by the video signal at a reduced resolution.
 7. Themethod of claim 6 wherein a video camera provides four video outputsignals that together comprise the image conveyed by the video signal.8. A system for detecting temporal misalignment in video frames that aredivided for transport, the system comprising: a signal divider fordividing a single signal S into N signal portions S₁ . . . S_(N); eachsignal portion for carrying a portion of the same video frame; Ntransport links, each link for receiving a respective signal Sx from thesignal divider and for delivering a signal Sx′ to a signal reassembler(1≦x≦N); the signal reassembler assembling signal portions S₁′ . . .S_(N)′ corresponding to S₁ . . . S_(N) into a single signal S′; and,wherein a misalignment monitor utilizes information inherent in thesignal portions S₁′ . . . S_(N)′ to detect whether temporal framemisalignment has occurred.
 9. The system of claim 8 further comprising:a transport block; and, the transport block including the transportlinks and signal processing; wherein communications between themisalignment monitor and the transport block are used to makeadjustments in the timing of frames.
 10. The system of claim 9 whereinthe signals are SDI signals for transporting 4K video.
 11. The system ofclaim 10 wherein: the misalignment monitor uses average picture level indetermining whether temporal frame misalignment has occurred.
 12. Thesystem of claim 8 further comprising: a signal source; whereincommunications between the misalignment monitor and the signal sourceare used to make adjustments in the timing of frames.
 13. The system ofclaim 12 wherein the signal source includes a video camera.
 14. Thesystem of claim 13 wherein the signals are Serial Digital Interfacesignals for transmitting 4K video.
 15. The system of claim 14 wherein:the misalignment monitor uses average picture level in determiningwhether temporal frame misalignment has occurred.
 16. A system fordetecting temporal misalignment in video frames that are divided fortransport, the system comprising: a signal source that conveys a videoimage using four 4K SDI video streams; transport links for transportingsignals derived from the four streams; and, a misalignment monitor thatuses average picture level to determine whether temporal framemisalignment involving the four streams has occurred during transport;wherein communications between the misalignment monitor and one of thesignal source and the transport links are used to make adjustments inthe timing of frames when temporal frame misalignment is discovered.